{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"../../img/ods_stickers.jpg\">\n",
    "## Открытый курс по машинному обучению. Сессия № 2\n",
    "</center>\n",
    "Автор материала: программист-исследователь Mail.ru Group, старший преподаватель Факультета Компьютерных Наук ВШЭ Юрий Кашницкий. Материал распространяется на условиях лицензии [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). Можно использовать в любых целях (редактировать, поправлять и брать за основу), кроме коммерческих, но с обязательным упоминанием автора материала."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Занятие 8. Разреженные данные, онлайн-обучение</center>\n",
    "## <center>Часть 2. Классификация отзывов к фильмам с SVM и логистической регрессией</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_files\n",
    "from sklearn.feature_extraction.text import (\n",
    "    CountVectorizer,\n",
    "    TfidfTransformer,\n",
    "    TfidfVectorizer,\n",
    ")\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Загрузим данные [отсюда](https://yadi.sk/d/Tg1Tflur333iLr). В обучающей и тестовой выборках по 12500 тысяч хороших и плохих отзывов к фильмам.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# поменяйте путь к файлу\n",
    "reviews_train = load_files(\n",
    "    \"/Users/y.kashnitsky/Yandex.Disk.localized/ML/data/imdb_reviews/train/\"\n",
    ")\n",
    "text_train, y_train = reviews_train.data, reviews_train.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of documents in training data: %d\" % len(text_train))\n",
    "print(np.bincount(y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# поменяйте путь к файлу\n",
    "reviews_test = load_files(\n",
    "    \"/Users/y.kashnitsky/Yandex.Disk.localized/ML/data/imdb_reviews/test/\"\n",
    ")\n",
    "text_test, y_test = reviews_test.data, reviews_test.target\n",
    "print(\"Number of documents in test data: %d\" % len(text_test))\n",
    "print(np.bincount(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Пример отзыва и соответствующей метки.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_train[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[1]  # плохой отзыв"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_train[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[2]  # хороший отзыв"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Идея \"мешка слов\"**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../img/bag_of_words.svg\" width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Простой подсчет слов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Составим словарь всех слов с помощью CountVectorizer.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer()\n",
    "cv.fit(text_train)\n",
    "\n",
    "len(cv.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Посмотрим на примеры полученных \"слов\" (лучше их называть токенами). Видим, что многие важные этапы обработки текста мы тут пропустили.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cv.get_feature_names()[:50])\n",
    "print(cv.get_feature_names()[50000:50050])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Закодируем предложения из текстов обучающей выборки индексами входящих слов. Используем разреженный формат.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = cv.transform(text_train)\n",
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Посмотрим, как преобразование подействовало на одно из предложений.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text_train[19726])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[19726].nonzero()[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[19726].nonzero()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Преобразуем так же тестовую выборку.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = cv.transform(text_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Обучим логистическую регрессию и линейный SVM.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "logit = LogisticRegression(n_jobs=-1, random_state=7)\n",
    "logit.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "svm = LinearSVC(random_state=7)\n",
    "svm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Посмотрим на доли правильных ответов на обучающей и тестовой выборках.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "round(logit.score(X_train, y_train), 3), round(svm.score(X_train, y_train), 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "round(logit.score(X_test, y_test), 3), round(svm.score(X_test, y_test), 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize_coefficients(classifier, feature_names, n_top_features=25):\n",
    "    # get coefficients with large absolute values\n",
    "    coef = classifier.coef_.ravel()\n",
    "    positive_coefficients = np.argsort(coef)[-n_top_features:]\n",
    "    negative_coefficients = np.argsort(coef)[:n_top_features]\n",
    "    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])\n",
    "    # plot them\n",
    "    plt.figure(figsize=(15, 5))\n",
    "    colors = [\"red\" if c < 0 else \"blue\" for c in coef[interesting_coefficients]]\n",
    "    plt.bar(np.arange(2 * n_top_features), coef[interesting_coefficients], color=colors)\n",
    "    feature_names = np.array(feature_names)\n",
    "    plt.xticks(\n",
    "        np.arange(1, 1 + 2 * n_top_features),\n",
    "        feature_names[interesting_coefficients],\n",
    "        rotation=60,\n",
    "        ha=\"right\",\n",
    "    );"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_grid_scores(grid, param_name):\n",
    "    plot(\n",
    "        grid.param_grid[param_name],\n",
    "        grid.cv_results_[\"mean_train_score\"],\n",
    "        color=\"green\",\n",
    "        label=\"train\",\n",
    "    )\n",
    "    plot(\n",
    "        grid.param_grid[param_name],\n",
    "        grid.cv_results_[\"mean_test_score\"],\n",
    "        color=\"red\",\n",
    "        label=\"test\",\n",
    "    )\n",
    "    legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_coefficients(logit, cv.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_coefficients(svm, cv.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Подберем коэффициент регуляризации для логистической регрессии.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "text_pipe_logit = make_pipeline(\n",
    "    CountVectorizer(), LogisticRegression(n_jobs=-1, random_state=7)\n",
    ")\n",
    "\n",
    "text_pipe_logit.fit(text_train, y_train)\n",
    "print(text_pipe_logit.score(text_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid_logit = {\"logisticregression__C\": np.logspace(-5, 0, 6)}\n",
    "grid_logit = GridSearchCV(text_pipe_logit, param_grid_logit, cv=3, n_jobs=-1)\n",
    "\n",
    "grid_logit.fit(text_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Лучшее значение C и соответствующее качество на кросс-валидации.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_logit.best_params_, grid_logit.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_grid_scores(grid_logit, \"logisticregression__C\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**То же самое для LinearSVC.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "text_pipe_svm = make_pipeline(CountVectorizer(), LinearSVC(random_state=7))\n",
    "\n",
    "text_pipe_svm.fit(text_train, y_train)\n",
    "print(text_pipe_svm.score(text_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "param_grid_svm = {\"linearsvc__C\": np.logspace(-5, 0, 6)}\n",
    "grid_svm = GridSearchCV(text_pipe_svm, param_grid_svm, cv=3, n_jobs=-1)\n",
    "\n",
    "grid_svm.fit(text_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_svm.best_params_, grid_svm.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_grid_scores(grid_svm, \"linearsvc__C\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_coefficients(\n",
    "    grid_svm.best_estimator_.named_steps[\"linearsvc\"],\n",
    "    grid_svm.best_estimator_.named_steps[\"countvectorizer\"].get_feature_names(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "На валидационной выборке:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_logit.score(text_test, y_test), grid_svm.score(text_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_pipe = make_pipeline(TfidfVectorizer(), LinearSVC())\n",
    "\n",
    "param_grid = {\"linearsvc__C\": np.logspace(-3, 2, 6)}\n",
    "grid_tfidf = GridSearchCV(tfidf_pipe, param_grid, cv=3, n_jobs=-1)\n",
    "grid_tfidf.fit(text_train, y_train)\n",
    "plot_grid_scores(grid_tfidf, \"linearsvc__C\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_coefficients(\n",
    "    grid_tfidf.best_estimator_.named_steps[\"linearsvc\"],\n",
    "    grid_tfidf.best_estimator_.named_steps[\"tfidfvectorizer\"].get_feature_names(),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_tfidf.best_score_, grid_tfidf.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## N-граммы"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "text_pipe = make_pipeline(CountVectorizer(), LinearSVC())\n",
    "\n",
    "param_grid = {\"linearsvc__C\": [0.01, 0.1, 1], \"countvectorizer__ngram_range\": [(1, 2)]}\n",
    "\n",
    "grid_bigram = GridSearchCV(text_pipe, param_grid, cv=3)\n",
    "\n",
    "grid_bigram.fit(text_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_grid_scores(grid_bigram, \"linearsvc__C\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_bigram.best_score_"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
